Make compiler requests safe to run repeatedly and in parallel - #8864

This makes compiler requests safe to run repeatedly, and on several OCaml domains at once, inside one process. bsc keeps its behaviour. The OCaml build system (#8780) will use this to compile in-process instead of starting bsc for every file (#8781), which is faster than Rust rewatch. The change is split out of #8781 so it can be reviewed on its own.

Changes

• Request-scoped state. The compiler's process-wide mutable state moves into domain-local, request-scoped state: Clflags, Js_config, warnings, environment and CRC caches, predefined types, Ident stamps, printers, diagnostics and backend caches. Almost all of it lives in Request_local keys:

  • every request starts with fresh values, as a new bsc process would;
  • the previous values are restored afterwards;
  • keys initialize in registration order.

  Only Ident stamps, the request context, output capture and the order-sensitive predefined types keep explicit handling. Clflags.reset is the single definition of bsc's defaults.

• Shared driver. The driver moves from rescript_compiler_main.ml into the rescript_compiler_driver library. bsc runs through it, and an embedding process can call run_request with its own working directory and its own runner for external commands such as PPXs. A relative -bs-project-root is resolved against that directory.

• Output capture. An embedded request's output is captured in memory, without temporary files. On Windows the captured text gets bsc's CRLF line endings, without doubling those a PPX already wrote. Output written before an escaping exception is kept and reported with it. The standalone bsc still writes directly to its stdout and stderr.

• Fixes found along the way:

  • races and leaked state that a single process never hit: Js_analyzer's free-variable accumulator was shared, so Js_shake could drop a live function definition; also Parmatch.Conv's name counter, Ctype's abbreviation memo and levels, and the parser hooks for error messages;
  • a failing or interrupted PPX no longer leaves its temporary AST files behind, and they have absolute paths when the temporary directory is relative;
  • a failing PPX's stderr no longer shifts the indentation of the error that follows it;
  • readers of .cmj, .ast and PPX output close their file when decoding fails, writers of .ast, .cmj and .cmi close it when writing fails, and .cmi files are opened close-on-exec;
  • -e's temporary source file is absolute when the temporary directory is relative, and is removed when compiling it fails;
  • bsc -verbose prints a PPX's command before the PPX's own output again.

Cost for bsc

Request-scoped state adds a domain-local lookup per access. For the standalone bsc this is not measurable: in clean builds of tests/tests (629 modules) through Rust rewatch, 9 interleaved runs per binary, the minimum CPU time was 12.66 s on master and 12.70 s with this PR, and the medians 13.49 s and 13.37 s.

ThreadSanitizer

An OCaml 5.5.0 ThreadSanitizer build of the in-process build system reported no data races in compiler code, on clean builds of rewatch/testrepo (4 domains), tests/tests (2, 8 and 12 domains), the gentype example and the React PPX tests. Two reports appeared in some tests/tests runs, both inside the runtime's parallel minor GC (oldify_one against pool_allocate). With the shared Js_analyzer accumulator restored, the same build reports 91 races, all in Js_analyzer, so the tool does catch this class of bug.

Tests

tests/ounit_tests/ounit_compiler_request_tests.ml checks that each kind of request-scoped state is fresh in every request and isolated between consecutive and concurrent requests, including:

• captured output and its line endings;
• the abbreviation memo, and the type-node counter (through the .cmt);
• generated names;
• artifact readers closing their file when decoding fails;
• the initialization order of Request_local keys.

🤖 Generated with Claude Code